ReproHack @ Leiden University

A hands-on Reproducibility Hackathon



📅 30th November, 2019
⏰ 10 am - 5pm
📍 Leiden University Library

Welcome 👋

What’s a ReproHack?

We are all excited by the progress made by many authors to make their papers reproducible by publishing associated code and data.

We know how challenging it can be so we want to showcase the value of the practice, both for original authors and as a learning experience for those who attempt to reproduce the work.

During a ReproHack, participants attempt to reproduce published research of their choice from a list of proposed papers with publicly available associated code and data. Participants get to work with other people’s material in a low pressure environment and record their experiences on a number of key aspects, including reproducibility, transparency and reusability of materials. At the end of the day we regroup, share our experiences and give feedback to the authors.


It’s imperative to note that ReproHacks are by no means an attempt to criticise or discredit work. We see reproduction as beneficial scientific activity in itself, with useful outcomes for authors and valuable learning experiences for the participants and the research community as a whole.

We strive to make this event open and inclusive to all. As such we ask you to read our Code of Conduct. By participating, you are expected to uphold this code.


Join the event and reproduce ♻️

Register and save your spot by clicking here!

Join us at the ReproHack and get working with other people’s material.

Benefits to participants:
  • Practical experience in reproducibility with real published materials and the opportunity to explore different tools and strategies.

  • Inspiration from working with other people’s code and data.

  • An appreciation that reproducibility is non trivial but that opening up your work for more people to engage with is the best way to help improve it. An appreciation that reproducibility has community value beyond just the validation of the results. For example, access to such materials increases the potential for reuse and understanding of the work.

Benefits to the whole research community:
  • Assessment of how reproducible papers are ‘out of the box’.

  • Evaluation of how successful current practices are and for what purpose.

  • Identification of what works and where the most pressing weaknesses in our approaches are.




Propose a paper 📜

Click here to nominate your paper!

You’ve put a lot of effort into making your work reproducible. Now let people learn from and engage with it!

We invite nominations for papers that have both associated code and data publicly available. We also encourage analyses based on open source tools as we cannot guarantee participants will have access to specialised licenced software.


Benefits to authors:
  • Feedback on the reproducibility of your work.

  • Appreciation for your efforts in making your work reproducible.

  • Opportunity to engage others with your research.


Proposed papers:


1. Supercurrent-induced Majorana bound states in a planar geometry

NA

submitted by André Melo

Why should we attempt to reproduce this paper?

We’ve tried to make it as easy as possible to reproduce. There’s some fun physics on the paper and it’s all done with Python!

Paper URL: https://dx.doi.org/10.21468/SciPostPhys.7.3.039

Data URL: https://doi.org/10.5281/zenodo.2653483

Code URL: https://doi.org/10.5281/zenodo.2653483

Useful programming skills: Python

2. Social-evaluative threat: Stress response stages and influences of biological sex and neuroticism

Poppelaars, E. S., Klackl, J., Pletzer, B., Wilhelm, F. H., & Jonas, E. (2019b). Social-evaluative threat: Stress response stages and influences of biological sex and neuroticism. Psychoneuroendocrinology, 109, 104378. https://doi.org/10.1016/j.psyneuen.2019.104378

submitted by Eefje S. Poppelaars

Why should we attempt to reproduce this paper?

This is a small dataset with a lot of missing data, so it’s quite challenging to produce reliable results. It uses multiple imputation to fill the missing data, so it would be interesting to see whether the results hold up when this is redone. However, since the multiple imputation takes a couple of hours to run (on a decent laptop), the final multiply imputed data is also included. Additionally, multiply imputed data needs a different statistical analysis approach, which you can get familiar with.

Paper URL: https://doi.org/10.1016/j.psyneuen.2019.104378

Data URL: https://doi.org/10.1016/j.dib.2019.104645

Code URL: http://dx.doi.org/10.17632/7vj8r76s6f.10

Useful programming skills: R

3. Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective

Luijken, K., Groenwold, R. H., Van Calster, B., Steyerberg, E. W., & van Smeden, M. (2019). Impact of predictor measurement heterogeneity across settings on the performance of prediction models: A measurement error perspective. Statistics in medicine.

submitted by Kim Luijken

Why should we attempt to reproduce this paper?

This methodological paper contains explanations and recommendations based on simulation studies. It is my first attempt at making a paper reproducible. I would be interested in knowing whether reproduced results are equal (or similar enough) and whether the code is user friendly.

Paper URL: https://onlinelibrary.wiley.com/doi/full/10.1002/sim.8183

Data URL: https://github.com/KLuijken/Prediction_Measurement_Heterogeneity_Predictor

Code URL: https://github.com/KLuijken/Prediction_Measurement_Heterogeneity_Predictor

Useful programming skills: R

4. Improving on Adjusted R-Squared

Karch, J. (2019, September 16). Improving on Adjusted R-Squared. https://doi.org/10.31234/osf.io/v8dz5

submitted by Julian D. Karch

Why should we attempt to reproduce this paper?

First, I think I came quite far in making it reproducible. The paper uses a simulation study. All the steps from the raw results of the simulation study to the final manuscript can be reproduced by clicking one button. This works because all code + all dependencies are stored online within a virtual machine that anybody can access. So, I think it is a quite good example for others to learn from. Second, the code running the simulation study itself is not included in this process because it would take far too long to run on one machine. I ran the code on a supercomputer. I would be interested in how people would try to reproduce such long-running code and whether they have feedback on how to improve sharing such long-running code.

Paper URL: https://doi.org/10.31234/osf.io/v8dz5

Data URL: https://doi.org/10.24433/CO.8023088.v1

Code URL: https://doi.org/10.24433/CO.8023088.v1

Useful programming skills: R

5. Spectral measure of color variation of black-orange-black (BOB) pattern in small parasitoid wasps (Hymenoptera: Scelionidae), a statistical approach

NA

submitted by Marcela Alfaro Córdoba

Why should we attempt to reproduce this paper?

PLOS one ensures that sharing code and data is enough for reproducibility. I think is not, and I want your opinion on giving them more specific recommendations based on what I did wrong in our article.

Paper URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6812806/#!po=0.467290

Data URL: https://github.com/malfaro2/Mora_et_al

Code URL: https://github.com/malfaro2/Mora_et_al

Useful programming skills: R

6. Use of significance test logic by scientists in a novel reasoning task

Morey and Hoekstra (in preparation)

submitted by Richard Morey

Why should we attempt to reproduce this paper?

The format of the paper is a bit unusual: it is contained, and compiled as, an R package. Although this would seem, on its face, to make it easier to reproduce, it is an open question how obvious it will be. I wonder to what extent people reproducing the results would prefer this to simple R scripts.

Paper URL: https://richarddmorey.github.io/Morey_Hoekstra_StatCognition/extras/manuscript.pdf

Data URL: https://github.com/richarddmorey/Morey_Hoekstra_StatCognition

Code URL: https://github.com/richarddmorey/Morey_Hoekstra_StatCognition

Useful programming skills: R

7. Cell Contractility Facilitates Alignment of Cells and Tissues to Static Uniaxial Stretch

Rens, E. G., & Merks, R. M. H. (2017). Cell Contractility Facilitates Alignment of Cells and Tissues to Static Uniaxial Stretch. Biophysical Journal, 112(4), 755–766.

submitted by Roeland Merks

Why should we attempt to reproduce this paper?

This paper shows a fun and interesting simulation result. I find it (of course) very important that our results are reproducible. In this paper, however, we did not include the exact code for these specific simulations, but the results should be reproducible using the code of our previous paper in PLOS Computational Biology (Van Oers, Rens et al. https://doi.org/10.1371/journal.pcbi.1003774). I am genuinely curious to see if there is sufficient information for the Biophys J paper or if we should have done better.

Other people have already successfully built upon the 2014 (PLOS) paper using our code; see e.g., https://journals.aps.org/pre/abstract/10.1103/PhysRevE.97.012408 and https://doi.org/10.1101/701037).

Paper URL: http://doi.org/10.1016/j.bpj.2016.12.012

Data URL: NA

Code URL: https://doi.org/10.1371/journal.pcbi.1003774.s006

Useful programming skills: C and Matlab for visualization

8. Algorithm configuration data mining for CMA evolution strategies

Sander van Rijn, Hao Wang, Bas van Stein, and Thomas Bäck. 2017. Algorithm configuration data mining for CMA evolution strategies. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’17). ACM, New York, NY, USA, 737-744. DOI: https://doi.org/10.1145/3071178.3071205

submitted by Sander van Rijn

Why should we attempt to reproduce this paper?

The original data took quite a while to produce for a previous paper, but for this paper, all tables and figures should be exactly reproducible by simply running the jupyter notebook.

Paper URL: https://dl.acm.org/citation.cfm?id=3071205

Data URL: https://github.com/sjvrijn/cma-es-configuration-data-mining/tree/master/arff

Code URL: https://github.com/sjvrijn/cma-es-configuration-data-mining

Useful programming skills: Python

9. Don’t Hold My Data Hostage - A Case For Client Protocol Redesign

Mark Raasveldt and Hannes Mühleisen: Don’t Hold My Data Hostage - A Case For Client Protocol Redesign, 43rd International Conference on Very Large Data Bases (VLDB2017)

submitted by Hannes Mühleisen

Why should we attempt to reproduce this paper?

We made a huge effort to ensure the paper is reproducible. But is it?

Paper URL: https://hannes.muehleisen.org/p852-muehleisen.pdf

Data URL: https://zenodo.org/record/1305845

Code URL: https://github.com/Mytherin/Protocol-Benchmarks

Useful programming skills: Python

10. Spatial modelling of rice yield losses in Tanzania due to bacterial leaf blight and leaf blast in a changing climate

Spatial modelling of rice yield losses in Tanzania due to bacterial leaf blight and leaf blast in a changing climate. C. Duku, A. H. Sparks, S. J. Zwart. Climatic Change 135.3-4 (2016) pp. 569–583. Springer Nature. doi: 10.1007/s10584-015-1580-2

submitted by Adam Sparks

Why should we attempt to reproduce this paper?

This was my third attempt at making a paper fully reproducible. To date I it’s the most reproducible that I have published. I’m interested to know what stumbling blocks exist that I’m not aware of (aside from needing software like ArcGIS to fully rerun the complete analysis).

Paper URL: https://link.springer.com/article/10.1007/s10584-015-1580-2?wt_mc=internal.event.1.SEM.ArticleAuthorOnlineFirst

Data URL: https://figshare.com/articles/MICORDEA/1408501

Code URL: https://github.com/adamhsparks/MICCORDEA

Useful programming skills: R, Python, ArcGIS

11. Climate change may have limited effect on global risk of potato late blight.

Sparks, A. H., Forbes, G. A, Hijmans, R. J., & Garrett K. A. (2014). Climate change may have limited effect on global risk of potato late blight. Global Change Biology, doi:10.1111/gcb.12587.

submitted by Adam Sparks

Why should we attempt to reproduce this paper?

This is a two-for one. The repository contains code for companion papers, the model development and the model implementation and analysis. As the repository notes, some data are not freely available so I’ve made an effort to allow the paper to be replicated as best possible with what’s available.

Paper URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/gcb.12587

Data URL: https://figshare.com/articles/Supporting_files_for_Climate_change_may_have_limited_effect_on_global_risk_of_potato_late_blight/1066070

Code URL: https://github.com/adamhsparks/Global-Late-Blight-MetaModelling

Useful programming skills: R

12. Sea level regulated tetrapod diversity dynamics through the Jurassic/Cretaceous interval

Tennant, J. P., Mannion, P. D., & Upchurch, P. (2016). Sea level regulated tetrapod diversity dynamics through the Jurassic/Cretaceous interval. Nature Communications, 7, 12737.

submitted by Jon Tennant

Why should we attempt to reproduce this paper?

Because it’s a fun paper, involving dinosaurs! But one which I myself have also attempted to reproduce in the past, and struggled with. There are a few additional tweaks that might throw some people off too.

Paper URL: https://www.nature.com/articles/ncomms12737

Data URL: https://www.nature.com/articles/ncomms12737#supplementary-information

Code URL: https://www.nature.com/articles/ncomms12737#supplementary-information

Useful programming skills: R, Perl

13. Evaluation of App-Embedded Disease Scales for Aiding Visual Severity Estimation of Cercospora Leaf Spot of Table Beet

Del Ponte EM, Nelson SC, Pethybridge SJ (2019) Evaluation of App-Embedded Disease Scales for Aiding Visual Severity Estimation of Cercospora Leaf Spot of Table Beet. Plant disease 103:1347-1356. 10.1094/PDIS-10-18-1718-RE

submitted by Emerson M. Del Ponte

Why should we attempt to reproduce this paper?

There are data and code written in RMarkdown which allows to reproduce the entire analysis and plots shown of the paper. It also allows to generate HTML document, which is a nice interface that facilitates the reader to understand better why some procedures were adopted and how to run them.

Paper URL: https://apsjournals.apsnet.org/doi/10.1094/PDIS-10-18-1718-RE

Data URL: https://osf.io/ezxps/

Code URL: https://github.com/emdelponte/paper-estimate-app

Useful programming skills: R

14. Bayesian determination of the effect of a deep eutectic solvent on the structure of lipid monolayers

Phys. Chem. Chem. Phys., 2019,21, 6133-6141

submitted by Andrew McCluskey

Why should we attempt to reproduce this paper?

I believe this represents the only example of a reproducible paper from scattering data collected at Diamond Light Source (UK) and the Institute Laue-Langevin (France)

Paper URL: https://doi.org/10.1039/c9cp00203k

Data URL: https://doi.org/10.15125/BATH-00548

Code URL: https://doi.org/10.5281/zenodo.2577796

Useful programming skills: Python, make

15. A multiscale Bayesian inference approach to analyzing subdiffusion in particle trajectories

K. Hinsen and G.R. Kneller, J. Chem. Phys. 145, 151101 (2016)

submitted by Konrad Hinsen

Why should we attempt to reproduce this paper?

This is one of the very few papers in biomolecular simulation for which code and data are available and which should be reproducible. But it is also three years old, so it is an interesting test case for the longevity of reproducible research. The infrastructure software is available at http://www.activepapers.org/python-edition/ (with instructions for installation and use)

Paper URL: https://doi.org/10.1063/1.4965881

Data URL: https://doi.org/10.5281/zenodo.162171

Code URL: https://doi.org/10.5281/zenodo.162171

Useful programming skills: Python

16. Resolving the Measurement Uncertainty Paradox in Ecological Management

Memarzadeh, M., & Boettiger, C. (2019). Resolving the Measurement Uncertainty Paradox in Ecological Management. The American Naturalist, 193(5). https://doi.org/10.1086/702704

submitted by Carl Boettiger

Why should we attempt to reproduce this paper?

This will probably be a non-trivial example to reproduce, owing to: (1) long-running code, (2) dependency on external data sources, (3) possibly challenging software dependencies – both trivial ones (e.g. setting up custom fonts and plot themes) and critical ones (requires an external R package wrapping a C++ algorithm, not available on CRAN and can sometimes have interesting compiler issues, like when Apple decided to break the clang compiler in 10.0). Ideally one could just run the R code given in the appendix on your local R session, but that may take a bit of effort. We’ve tried to take steps to address those issues by providing caches of slow-running parts, providing a docker container, and providing sufficient annotations, but who knows!

Paper URL: https://doi.org/10.1086/702704

Data URL: NA

Code URL: https://github.com/boettiger-lab/pomdp-intro

Useful programming skills: R

17. Comparisons of Citizen Science Data-Gathering Approaches to Evaluate Urban Butterfly Diversity

Prudic KL, Oliver JC, Brown BV, Long EC. 2018. Comparisons of Citizen Science Data-Gathering Approaches to Evaluate Urban Butterfly Diversity. Insects. 9(4):E186. doi: 10.3390/insects9040186

submitted by Kathleen Prudic

Why should we attempt to reproduce this paper?

This is a fairly digestible paper with statistical analyses and data visualization that rely heavily on open data from citizen science projects.

Paper URL: https://doi.org/10.3390/insects9040186

Data URL: https://doi.org/10.5281/zenodo.1436741

Code URL: https://doi.org/10.5281/zenodo.1436741

Useful programming skills: R

18. Bivariate spatial point patterns in the retina: a reproducible review.

Eglen SJ (2016) Bivariate spatial point patterns in the retina: a reproducible review. Journal de la Société Française de Statistique 157:33–48.

submitted by Stephen Eglen

Why should we attempt to reproduce this paper?

Tell me what I should improve!

Paper URL: https://github.com/sje30/eglen2015

Data URL: NA

Code URL: NA

Useful programming skills: R

19. A data repository and analysis framework for spontaneous neural activity recordings in developing retina

https://doi.org/10.1186/2047-217X-3-3

submitted by Stephen Eglen

Why should we attempt to reproduce this paper?

Tell me what I can improve on; maybe think of other visualisations for data?

Paper URL: https://doi.org/10.1186/2047-217X-3-3

Data URL: NA

Code URL: http://www.damtp.cam.ac.uk/user/eglen/waverepo/

Useful programming skills: R

20. Population structure and phenotypic variation of Sclerotinia sclerotiorum from dry bean (Phaseolus vulgaris) in the United States

Kamvar ZN, Amaradasa BS, Jhala R, McCoy S, Steadman JR, Everhart SE. 2017. Population structure and phenotypic variation of Sclerotinia sclerotiorum from dry bean (Phaseolus vulgaris) in the United States. PeerJ 5:e4152 https://doi.org/10.7717/peerj.4152

submitted by Zhian N. Kamvar

Why should we attempt to reproduce this paper?

This paper is reproduced weekly in a docker container on continuous integration, but it is also set up to work via local installs as well. It would be interesting to see if it’s reproducible with a human operator who knows nothing of the project or toolchain.

Paper URL: https://peerj.com/articles/4152/

Data URL: https://osf.io/k8wtm

Code URL: https://github.com/everhartlab/sclerotinia-366

Useful programming skills: R, Make and knowledge of Docker containers

21. PREPRINT: Using digital epidemiology methods to monitor influenza-like illness in the Netherlands in real-time: the 2017-2018 season

Schneider P, Van Gool C, Spreeuwenberg P, Hooiveld M, Donker GA, Barnett DJ, Paget J. Using digital epidemiology methods to monitor influenza-like illness in the Netherlands in real-time: the 2017-2018 season. BioRxiv. 2018 Jan 1:440867.

submitted by Paul Schneider

Why should we attempt to reproduce this paper?

This preprint is an attempt to reproduce Google Flu Trend in the Netherlands.

The whole paper + code is meant to be easily reproducible and transferable to other countries and/or areas. If you are familiar with time series data, lasso regression and cross validation, the analysis should be straight forward.

If anyone is interested, I could also provide influenza data for other European countries.

Paper URL: https://www.biorxiv.org/content/10.1101/440867v1.full

Data URL: https://zenodo.org/record/1459862#.XQbNIG8vNPM

Code URL: https://zenodo.org/record/1459862#.XQbNIG8vNPM

Useful programming skills: R

22. Growth Dynamics of Independent Gametophytes of Pleurosoriopsis makinoi ( Polypodiaceae)

Bulletin of the National Museum of Nature and Science, Series B (Botany) 45: 77–86

submitted by Joel Nitta

Why should we attempt to reproduce this paper?

It uses the drake R package that should make reproducibility of R projects much easier (just run make.R and you’re done). However, it does depend on very specific package versions, which are provided by the accompanying docker image.

Paper URL: https://www.joelnitta.com/publication/2019-03-27_pleurosoriopsis/

Data URL: https://github.com/joelnitta/pleurosoriopsis

Code URL: https://github.com/joelnitta/pleurosoriopsis

Useful programming skills: R

23. Open Trade Statistics

@misc{open_trade_statistics_2019, title = {OTS BETA DASHBOARD}, url = {https://shiny.tradestatistics.io/}, author = {{Open Trade Statistics}}, publisher = {Open Trade Statistics}, year = {2019}, month = {Apr}, note = {Accessed: June 22, 2019} }

submitted by Mauricio Vargas

Why should we attempt to reproduce this paper?

The focus of the project is reproducibility. Here we show the differences to access data compared to similar initiatives: https://ropensci.org/blog/2019/05/09/tradestatistics/. Also, similar projects have obscure parts, while our exposes the code from raw data downloading to dashboard creation.

Paper URL: https://shiny.tradestatistics.io

Data URL: https://api.tradestatistics.io

Code URL: https://github.com/tradestatistics

Useful programming skills: R, Shiny

24. model4you: An R Package for Personalised Treatment Effect Estimation

Seibold, H., Zeileis, A. and Hothorn, T., 2019. model4you: An R Package for Personalised Treatment Effect Estimation. Journal of Open Research Software, 7(1), p.17. DOI: http://doi.org/10.5334/jors.219

submitted by Heidi Seibold

Why should we attempt to reproduce this paper?

I guess it could be a cool learning experience. The paper is written with knitr, uses a seed, is part of the R package it describes, was openly written using version control (SVN, R-Forge) and is available in an open access journal (@up_jors).

Paper URL: http://doi.org/10.5334/jors.219

Data URL: NA

Code URL: https://r-forge.r-project.org/scm/viewvc.php/pkg/model4you/inst/JORS/?root=partykit

Useful programming skills: R, knitr (LaTeX), version control (SVN)

25. Comparing theory-driven and data-driven attractiveness models using images of real women’s faces

Holzleitner et al. (In press). Comparing theory-driven and data-driven attractiveness models using images of real women’s faces, JEP:HPP.

submitted by Ben Jones

Why should we attempt to reproduce this paper?

Complex analyses over multiple variables. In press, so we can still fix errors ahead of publication!!

Paper URL: https://psyarxiv.com/vhc5k

Data URL: https://osf.io/jurcq/

Code URL: https://osf.io/jurcq/

Useful programming skills: R


Program 📅

10:00 Coffee and Tea ☕

10:30 Welcome

10:40 Presentation about tools for Reproducible Research by Dr. Anna Krystalli

11:30 Forming groups and start hackathon 👨🏻‍💻 👨🏼‍💻

12:30 Lunch buffet 🥗 🥙

14:30 2nd presentation (tbc)

15:00 Continue hacking 👨🏽‍💻👩🏼‍💻

16:30 Drinks and bites 🍻